Ece 695 " Statistical Learning Theory " Lecture Notes Lecture 4 *

In the previous lectures, our set-up was as follows. We had a set X of elements and a distribution P over the set X × {0, 1}. The distribution defines the probability (or density) for each element x ∈ X to “come up”, and the probability for a certain property of x to hold or not to hold. If (x, 1) “comes up,” then x has the property, and if (x, 0) “comes up,” then x does not have the property. The learning problem consists in predicting, given x generated by P , whether or not this property holds. We are given an independent, identically distributed (i.i.d.) sample S ∈ (X × {0, 1}) of distribution P that contains m pairs (xi, yi), i = 1, 2, . . . ,m, where xi ∈ X and yi ∈ {0, 1}, and we use S to find a function h : X → {0, 1} that takes x and predicts y in a new pair (x, y). The prediction function h, called a hypothesis, must be selected from a given set H of functions, called the hypothesis class. Ideally, we should choose h ∈ H such that it minimizes the true error E (h) defined as E (h) := P [(x, y) ∼ P : h(x) 6= y] In practice, however, all we can do is to choose h ∈ H that minimizes the training error E(h) which equals